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As part of the Future Research Assessment Programme (FRAP), the four funding bodies (Research 


England, Scottish Funding Council, Higher Education Funding Council for Wales, Department for the 
Economy, Northern Ireland) are seeking views on the design of the UK’s future research assessment 
system. The consultation asks about: 


e The purposes of a future exercise 
e The principles that should guide its development 
e The assessment criteria and processes 


This document shares the NCCPE’s response, which was submitted on 6t May 2022. We would like 
to thank the many people who have contributed to this document. 


You can find full details of the consultation here. 


The document begins with a summary of our key suggestions, followed by the full response. 


Executive summary 


EVOLUTION NOT REVOLUTION 

We argue for evolution not revolution: the consultation invites ‘blue skies’ thinking about radical 
alternatives to the Research Excellence Framework (REF), but on balance we see value in 
continuing to run the REF. 


e The REF has initiated a significant re-orientation of research culture and practice and 
increased our collective focus on the value of engagement with society to realise public 
benefits from research. While we know it is far from perfect, and there are well 
documented negative effects, we see significant value in continuing to run the exercise 
and to build on its positive effects. We argue for reasoned adaptations to the REF, rather 
than radical changes. 

e The area where we argue for the most significant change is in relation to how the REF 
assesses research and impact culture. We think that this is the part of the REF that is 
functioning least effectively. 

e Wewould like to see formal recognition of the instrumental influence of the REF on 
capacity building and ‘system shaping’, and for this to be highlighted as a core purpose of 
the exercise (rather than a side effect). This will require significant re-working of the 
Environment section, which is where the bulk of our recommendations lie. 

e We highlight significant issues with how the REF treats Equality, Diversity and Inclusion 
(EDI) in relation to the assessment of impact, and argue that major changes are required 
to address this, in line with the Public Sector Equality Duty which puts the onus on 
organisations to positively promote equality, not just avoid discrimination. 

e = If the sophistication and rigour with which the REF assesses impact is enhanced as we 
have outlined, then we would like to see the weighting for the impact portfolio be raised 
from 25% to 30%, and Environment from 15% to 20% (and outputs reduced to 50%) in 
recognition of the importance of valuing the public benefit achieved through research, 
and the importance of accelerating enhancements to research culture. This change will 
focus attention on the vital need for HEls to invest in capability to support impact, 
including through their investment in public engagement expertise. 


ASSESSING IMPACT 


e We would like to see an evolution in the framing of ‘impact’: the term is a blunt 
instrument to describe the complex process of realising benefits from research and 
innovation. We recommend a change to ‘Public Benefit’ or ‘Social Benefit’. 

e We argue the case for an additional criterion for assessing impact case studies, to mirror 
the inclusion of ‘rigour’ in the assessment of research outputs. We think that it is 
important to focus attention on the professional care taken to realise impact, in particular 
on how EDI and ethics are taken into account, and how the approach aligns with good 
practice principles of engagement. We accept that some impacts are serendipitous, but it 
is imperative that the sector develops greater sophistication and sensitivity in how it takes 
account of the wider social and ethical context for research, and that this is reflected in 
the framing and assessment of the impact. This would enable the same kind of balanced 
assessment of methodological rigour and outcomes as is achieved in the evaluation of 
research outputs. One could argue for using the same term (‘rigour’) to focus attention on 
the underpinning professional practice, but we suggest that the criterion ‘sensitivity’ 
might work better to convey the expectation 

e We think there is a strong case for revisiting the criteria for inclusion of underpinning 
research in the impact case studies, with underpinning research judged only on the basis 


of its rigour, not its significance and originality. Robust research that isn’t particularly 
original or academically significant can have 4* impact. Anecdotally, it appears that 
nervousness about the 2* threshold currently used has led to HEls choosing not to submit 
excellent impact activity. 

e We argue that the decision for REF 2021 to incorporate the impact template in the 
Environment section should be reversed, and that a similar device to the impact template 
should be reinstated (for instance a narrative account of the underpinning approach to 
support impact, building perhaps on lessons learned from the use of narratives in the 
Local Growth and Public and Community Engagement in the KEF). This would 
contextualize the submitted impact case studies within the Unit’s wider impact and 
engagement approach. Anecdotally, it appears that few panels actually brought the 
impact section of the Environment statement to the attention of their impact assessors in 
REF 2021, potentially resulting in a significantly less rounded judgement. 

e As the sector matures in confidence in its approach to assessing impact, we would like to 
see greater openness and transparency in the judgements that panels make about impact 
(which is still something of a ‘black box’). We think that there is a case to be made for 
publishing the grades awarded to individual case studies, although we are aware that 
there are risks with this (for instance, how this is used in performance management within 
HEls, and might inhibit staff from coming forward with impact case studies). We can 
already work out the grade of many, so this is not a big change, and would enable much 
more effective analysis, whilst promoting good practice and avoiding poor examples being 
used as exemplars by others. Ideally, in addition to scores, the justification for a selection 
of particularly instructive scores should be published, as there may (for instance) be 
significant weaknesses in 4* cases that panels have picked up on, despite grading them 
4*. 

e We introduce a ‘thought experiment’ to consider the potential benefits of decoupling the 
assessment of outputs and impact, and perhaps sequencing these exercises so that they 
don’t happen simultaneously. If this were implemented, the assessment of impact could 
be delivered through a ‘lighter touch’ approach, with perhaps four broad discipline-linked 
panels and a fifth interdisciplinary panel. Each would be chaired by an impact specialist, 
and largely made up of impact specialists with a small number of ‘Research assessors’ to 
assure that the underpinning research meets the threshold. This would have a number of 
advantages: 

e Rebalancing the exercise so that Impact and Outputs have equivalent status 
e Strengthening the role of ‘users’ and stakeholders in research 

e More effectively bringing impact specialism into the review process 

e Better networking and connecting the impact expertise across the exercise 
e Reducing some of the overheads of the exercise 


We are also aware of the downsides of this approach: 
e It risks fuelling a two-tier approach in institutions 
e It risks downplaying the vital interaction between research and impact, and silo-ing 
the two processes 


ASSESING RESEARCH ENVIRONMENT 
We argue for significant changes to the Environment statement. 


e There is currently a great deal of ‘noise’ and burden in how we collectively incentivise, 
assess and monitor research and impact culture, with a host of different, overlapping 
instruments and frameworks. The Future Research Assessment Programme (FRAP) 
provides an opportunity to step back and simplify the landscape, and to articulate a 


clearer purpose and function for the environment section in the REF. To do so, we argue, 
will require a ‘design-led’ and ‘whole system’ approach, along the lines proposed by 
Universities UK in their work to rationalize the different research concordats. 

We question the ongoing value of the existing criteria of ‘vitality’ and ‘sustainability’ for 
assessing the research environment, as we are not confident that these adequately focus 
attention on the key drivers of research and impact culture. We suggest that the enabling 
principles described in UKRI’s new strategic framework provide more robust and 
productive focal points, targeting Diversity, Connectivity, Resilience and Engagement. We 
think work could be done to derive indicators for each of these principles to assess and 
compare performance. This would have the added benefit of encouraging common 
purpose and joining up thinking across the whole R&I system. 

We argue that evaluation of the research environment should take more account of the 
very significant place-based and institutional disparities which are currently hidden from 
view in the REF’s assessment approach. There isn’t a ‘level playing field’ in terms of the 
investment and capabilities that different HEls can draw upon to underpin their 
production of outputs and to invest in impact-generating activity. In the context of 
‘Levelling up’ and the increasing focus on place based and civic responsibility, we also 
argue for the inclusion of contextual information about a HEI’s physical location. We 
would like to see a separate, data-driven ‘Context’ section, which builds upon the lessons 
learned from developing clusters in the KEF, to help fairer judgements to be made about 
what institutions have achieved with the resources at their disposal. 


Section one: purposes of research assessment 
Preamble: framing comments from the consultation document 


The current assessment exercise serves three primary purposes: 
e inform the selective allocation of funding to HEIs for research; 
e provide accountability for public investment in research; and 
e provide benchmarking information. 


In addition, an independent review of REF 2014, carried out by Lord Stern in 2016 identified three 
further purposes: 
e provide an evidence base to inform strategic national priorities; 
e provide an evidence base for HEIs and other bodies to inform decisions on resource 
allocation; 
e create a performance incentive for HEIs. 


The funding bodies have set out their intention to retain the link between assessment outcomes and 
funding, and to require any future exercise to provide accountability for public investment in 
research. 


1. In addition to enabling the allocation of research funding and providing accountability for 
public investment in research, which purposes should a future UK research assessment exercise 
fulfil? Select all that apply. 


a) Provide benchmarking information 


b) Provide an evidence base to inform strategic national priorities 


c) Provide an evidence base for HEls and other bodies to inform decisions on 
resource allocation 


d) Create a performance incentive for HEIs. 


2. What, if any, additional purposes should be fulfilled by a future exercise? 


The REF (or its replacement) can fulfil all of the additional purposes listed above. However, we 
think it would be a mistake to overcomplicate things by creating a long list of additional purposes. 


The key purpose which we think should be added (in addition to the identified purposes of 
allocation of funding and providing accountability) is to build collective capability across the sector 
and contribute to the development of a high performing research ‘system’. 


It is widely accepted that the REF does already exert enormous influence on sector behaviour — 
largely because of the way that HEls chose to interpret and use it. The consequences can be 
damaging (driving competitive behaviour, distorting performance, creating a huge burden that 
outweighs the benefits), but it has also enabled a significant culture shift, helping to rebalance the 
research system to better meet the expectations of wider society, including through the 
encouragement of engagement with the public. We need to grip the REF as a way of building 
collective capacity and take more deliberate steps to exploit its positive influence in this area. 


So how does the REF currently support the building of capability and system ‘health’? 


e It enables us to review and refine professional standards, by providing a periodic 
opportunity for the sector to work together (and with users) to take stock of what quality 
looks like, to define descriptors of this and then to apply these. These standards also 
provide the basis for other forms of evaluation 

e It helps to showcase innovation and excellence in all areas of the current research system 
and is the only benchmarked overview of research performance and the state of the 
sector 


The broad thrust of our response is to outline how we think these benefits might be enhanced, in 
particular by significant changes to the Environment statement. 


3. Could any of the purposes be fulfilled via an alternative route? If yes, please provide 
further explanation. 


While we are keen to see significant changes to the next iteration of the REF, we believe that 
there is great value in maintaining continuity with previous iterations of the exercise. Continuing 
on the same trajectory will enable us to deepen our collective understanding of ‘what works’ and 
how to achieve excellent results. Therefore, we argue for evolution not revolution in our approach 
to research assessment. 


A key part of the evolution is to enhance the contribution of the REF to improving the functioning 
of the R&I system, balancing this against the need to hold the sector accountable and to award 
funding fairly and robustly. 


But the REF is only one of a range of different instruments that are used to influence and reward 
research culture and to build our collective ‘infrastructure’ to support excellence. The design of 
Future Research Assessment needs to take a holistic and systemic approach to aligning these 
different instruments and minimizing bureaucracy and ‘noise’ in the system. This applies in 
particular to the role of the Environment section. 


We list below some of the other initiatives that are underway. The revised Environment section 
should not duplicate the reporting required for these other instruments, and the solution should 
be guided by the principles of the bureaucracy Review (Harmonisation, Simplification, 
Proportionality, Flexibility etc): 


e The review of Concordats and Initiatives 

e — UKRI’s work to simplify and enhance its funding regulations (Simpler and Better Funding) 

e Research England’s KE review (which includes a review of the KE Concordat and the KEF) 
and which provides a complementary lens onto the ‘impact and engagement 
infrastructure’ in England 

* SFC’s review of Coherent Provision and of KE & | 

e The Government's People and Culture strategy and the ministerial working group 
overseeing this 

e The introduction of the narrative CV 

e The new Committee on Research Integrity 


4. Do you have any further comments to make regarding the purposes of a future 
research assessment system? 


While we argue for evolution, we don’t encourage complacency. Each iteration of the REF should 
set stretching new collective goals, while maintaining continuity with the past and paying careful 
attention to assessing the unintended or negative consequences of the exercise. The role of 
research in society is constantly evolving as new demands, new challenges and new opportunities 
impact on the sector. Research Assessment should be sensitive to these changes, and the periodic 
nature of the REF allows for useful collective stock taking and re-calibration of our ambition. 


Our particular interest is in how the REF can enable a step change in public engagement strategy 
and practice, and this informs the rest of our response. We found the publication of the REF 
Impact Case studies after REF 2014 an immensely useful data set, allowing us to review the ‘state 
of the art’ in public engagement with research, and to identify areas where enhancement could 
be realised. We await with interest the results of the next REF, which will allow us to reflect on the 
distance travelled in the sector’s approach to public engagement, and to identify areas where 
further sophistication and challenge might be built into the next iteration. 


Anticipating the results, we argue for developments in three key areas: 


e To bring in a more rigorous focus on ethics, values and risks (so that the same research 
integrity and ethical standards are applied to impact as to research) 

e To more robustly incentivise EDI, in particular in relation to the impacts realised beyond 
academia. 

e To sharpen the focus on the contribution of engagement and impact professionals and to 
better incentivise long term and strategic investment in building capacity in this area. 
Impact is a different skill set and requires reward and recognition separately from 
research skills — though they will share attributes in common. 


Section two: setting priorities 
Preamble 


The roundtable discussions identified a number of priorities and guiding principles that participants 
believed should drive the development of a future assessment exercise. Some of these principles 
potentially preclude or conflict with others. It is therefore important for the funding bodies to 
understand which principles the sector would like them to prioritise when designing a future system 


5. To what extent should the funding bodies be guided by the following considerations in 
developing the next assessment system? Please rank the considerations from 1 (most 
important) to 9 (least important) 


a) Ability of the system to promote research with wider socio-economic impact. | 4 


b) Comparability of assessment outcomes (across institutions, disciplines and/or | 2 
assessment exercises) 


c) Ensuring that the bureaucratic burden of the system is proportionate 


d) Impact of the assessment system on local/regional development 


e) Impact of the system on research culture 


f) Impact of the system on the UK research system’s international standing 


g) Maintaining continuity with REF 2021 


h) Providing early confirmation of the assessment framework and guidance 
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i) Robustness of assessment outcomes 


6. Relating to research culture, to what extent should the funding bodies be guided by the 
following considerations in developing the next assessment system? Please rank the 
considerations from 1 (most important) to 6 (least important) 


a) Impact of the assessment system on research careers 3 

b) Impact of the assessment system on equality, diversity and inclusion 

c) Ability of the assessment system to promote collaboration (across 5 
institutions, sectors and/or nations) 

d) Impact of the system on inter- and transdisciplinary research 4 

e) Impact of the system on open research 6 

f) Impact of the system on research integrity 2 


7. What, if any, further considerations should influence the development of a future assessment 
system? Please set out the considerations and indicate where they should be located in the list 
of priorities. 


We strongly support the importance of prioritising the impact on research culture, but this is a 
rather broad and woolly consideration. We think this could be better framed as the acceleration 
of capacity building and collaboration across the sector. 


We have indicated above that we think that the robustness and comparability of the assessment 
outcomes are both a top priority for Future Research Assessment. But we think that there are 
significant issues with comparability which need attention in the next exercise. 


The first concerns the way in which the REF is currently ‘blind’ to the unequal distribution of 
resources and capabilities across the R&I system. Small departments with a skeletal staff and 
minimal central support are competing against institutions who have benefited from a substantial 


long-term investment in their research and impact capability, with significant ‘top up’ funding 
from, for instance, Impact Acceleration Accounts, HEIF and donations. We argue that more 
account should be taken of the resources HEls have at their disposal and of what teams manage 
to achieve with what inputs. 


It is also now widely accepted that the REF is ‘place blind’, taking no account of the contexts 
within which HEls work to exchange and build knowledge. Given the increasing focus on ‘levelling 
up’, and on the civic contribution of HEls, there is a strong case to be made for the Institutional 
Environment statement (5a) to collect place-related data to help contextualise HEls’ performance 
and approach. We are aware of various approaches that are in development to identify data to 
describe and help differentiate between the contexts in which HEls operate, and on how these 
constrain and enable performance. 


It is interesting to note that the KEF has introduced clustering to allow meaningful comparisons to 
be drawn between different HEIs. Lessons could be applied from this exercise. We note the 
reservations about clustering raised by Conservatoires UK, and others. 


8. How can a future UK research assessment system best support a positive research culture? 


A positive research culture requires: 
-  Aset of shared principles that describe the characteristics of a positive research culture 
- Arobust but proportionate approach to holding institutions to account for their 
performance in relation to these principles 
- Adynamic learning and self-improvement culture that accelerates the adoption of good 
practice 


As noted above (Q.3), it will be vital that the next iteration of the REF intelligently aligns its criteria 
and focus with the host of other developments that are in train to refine, assess and promote 
changes to research culture. The Environment section of the REF provides an excellent route to 
assessing performance in this area, but quite how will only become clear as the other 
developments mature and coalesce. 


We suggest the following principles could helpfully be applied to the Environment section in the 
REF: 


- That it reinforces and aligns with other principles and criteria defining positive research 
culture, to minimise ‘noise’ in the system 

- That it doesn’t duplicate other evidence gathering and assessment processes, and 
addresses gaps in intelligence 

- That it creates significant ‘added value’ (for instance by aggregating data from several 
different sources, or by providing longitudinal insight) 

- That it accelerates the sharing of intelligence and evidence about effective practices 

- That it informs our understanding of how investment in research / impact culture can be 
best targeted to benefit the whole R&I system 

- That it draws on the significant evidence base about how to support culture change in 
engagement and impact. We would point to the NCCPE’s work in this area, along with a 
host of other useful tools and frameworks including the work of Bailey and Phipps on 
Impact Literacy and Impact health, and the work of Reed and Fazey on the determinants 
of impact culture. 


Section three: identifying research excellence 


Preamble: The funding bodies agreed that the outcomes of the next assessment framework should 
continue to enable them to allocate funding based on research excellence. It is therefore important 
that the exercise adequately captures those elements that constitute excellent research and assesses 
them robustly against appropriate criteria. 


Components of excellence The REF currently assesses three elements: 
e Outputs (60%) 
e Impact (25%) 
e Environment (15%) 


Roundtable discussions suggested that a broader definition of excellence, which recognises and 
rewards a wider range of activities and inputs, may better support a healthy, inclusive research 
system. Participants recognised that some of these elements are already captured through the 
environment statement but called for increased weighting and/or a more structured approach to 
assessing elements such as open research practices and policies to support equality, diversity and 
inclusion. At the same time, participants questioned how these components might be assessed 
robustly and consistently. It was agreed that robust indicators would be required and participants 
acknowledged that this may be challenging 


Components of research excellence 


9. Which of the following elements should be recognised and rewarded as components of 
research excellence in a future assessment exercise? 

(Multiple options: ‘Should be heavily weighted’ — ‘Should be moderately weighted’ — ‘Should be 
weighted less heavily’ — ‘Should not be assessed’ — ‘Don’t know’) 


a) Research inputs (e.g. research income, internal investment in Moderately 
research and in researchers) 

b) Research process (e.g. open research practices, collaboration, Moderately 
following high ethical standards) 

c) Outputs (e.g. journal articles, monographs, patents, software, Heavily 


performances, exhibitions, datasets) 


d) Academic impact (contribution to the wider academic community | Moderately 
through e.g. journal editorship, mentoring, activities that move the 
discipline forward) 


e) Engagement beyond academia Moderately 


f) Societal and economic impact Heavily 


g) Other (please specify). 


10. Do you have any further comments to make regarding the components of research 
excellence? 


It makes sense for the exercise to most heavily weight the outputs and impacts of research. But to 
maximise collective learning and development, then a focus on process and inputs is also vital. 


If the sophistication and rigour with which the REF assesses impact is enhanced along the lines we 
outline in our response, then we would like to see the weighting for the impact portfolio be 
raised from 25% to 30%, and Environment from 15% to 20% (and outputs reduced to 50%) in 
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recognition of the importance of valuing the public benefit achieved through research, and the 
importance of accelerating enhancements to research culture. This change will focus attention on 
the vital need for HEls to invest in capability to support impact, including through their investment 
in public engagement expertise. 


Assessment criteria 


In assessing submissions, the REF expert panels assess three distinct elements of each submission, 
against the following generic criteria: 


1. Outputs: The panels assess the quality of submitted research outputs in terms of their 
‘originality, significance and rigour’, with reference to international research quality 
standards. 


2. Impact: The panels assess the ‘reach and significance’ of impacts on the economy, society, 
culture, public policy or services, health, the environment or quality of life that were 
underpinned by excellent research conducted in the submitted unit. 


3. Environment: The panels assess the research environment of the submitted unit in terms of 
its ‘vitality and sustainability’, including the approach to enabling impact from its research, 
and its contribution to the vitality and sustainability of the wider discipline or research base. 


Roundtable discussions revealed some concerns about the wider effects of the current REF 
assessment criteria. For example, the criterion of ‘originality’ in relation to outputs was seen to 
discourage the submission of replication studies and reviews, which are essential to driving forward 
high-quality research. Similarly, the criterion of ‘reach’ for impact was seen to discourage the 
submission of research with local impacts, despite clarifications in the Panel Criteria that reach 
should not be interpreted in this way. 


11. Are the current REF assessment criteria for outputs clear and appropriate? Yes 
(Yes/No/Don’t know) 
e Originality 
e Significance 

e Rigour 


12. Do you have any further comments to make regarding the criteria for assessing 
outputs? 


13. Are the current REF assessment criteria for impact clear and appropriate? Yes 
(Yes/No/Don’t know) 
e Reach 
e Significance 


14. Do you have any further comments to make regarding the criteria for assessing 
impact? 
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We would like to see an evolution in the framing of ‘impact’: the term is a blunt instrument to 
describe the complex process of realising benefits from research and innovation. We recommend 
a change to ‘Public Benefit’ or ‘Social Benefit’. 


We have significant concerns about how the REF currently takes account of EDI in relation to 
impact, which we explain further in our response to the final question about metrics. 


We argue the case for an additional criterion for assessing impact case studies, to mirror the 
inclusion of ‘rigour’ in the assessment of research outputs. We think that it is important to focus 
attention on the professional care taken to realise impact, in particular on how EDI and ethics are 
taken into account, and how the approach aligns with good practice principles of engagement. 


We accept that some impacts are serendipitous, but it is imperative that the sector develops 
greater sophistication and sensitivity in how it takes account of the wider social and ethical 
context for research, and that this is reflected in the framing and assessment of the impact. This 
would enable the same kind of balanced assessment of methodological rigour and outcomes as is 
achieved in the evaluation of research outputs. One could argue for using the same term (‘rigour’) 
to focus attention on the underpinning professional practice, but we suggest that the criterion 
‘sensitivity’ might work better to convey the expectation 


We also think there is a strong case for revisiting the criteria for inclusion of underpinning 
research in the impact case studies, with underpinning research judged only on the basis of its 
rigour, not its significance and originality. Robust research that isn’t particularly original or 
academically significant can have 4* impact. Anecdotally, it appears that nervousness about the 
2* threshold as currently configured has led to HEls choosing not to submit excellent impact 
activity. 


15. Are the current REF assessment criteria for environment clear and No 
appropriate? (Yes/No/Don’t know) 
e Vitality 
e Sustainability 


16. Do you have any further comments to make regarding the criteria for assessing 
environment? 


We have reservations about how well the revised Environment statement worked in the last REF. 
We feel that this is the area where the most significant enhancements need to be made to 
maximise the ‘capacity building’ potential of the REF. Anecdotally, it appears that these 
statements were hard to assess, and that the quantity of information submitted meant that it was 
hard for assessors to ‘see the wood for the trees’. 


We question the value of the existing criteria of ‘vitality’ and ‘sustainability’ for assessing the 
research environment, as we are not confident that these adequately focus attention on the key 
drivers of research and impact culture. We suggest that the enabling principles described in UKRI’s 
new strategic framework provide more robust and productive focal points, targeting Diversity, 
Connectivity, Resilience and Engagement. We think work could be done to derive indicators for 
each of these principles to assess and compare performance. This would have the added benefit 
of encouraging common purpose and joining up thinking across the whole R&I system. 
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Examples of the kinds of data which might be required to demonstrate adherence to these 
principles include: 
e Information about the gender pay gap in institutions / Units 
e Information about FTE staff supporting public and other forms of external engagement 
and impact 


We also think it is important that the Environment section better recognises and rewards the 
diversity of practices and contexts for research across the sector. While ‘absolute standards’ of 
excellence can be applied to research outputs and impacts, the Environment section could work 
much harder to recognise the diversity of approaches and contexts in which institutions work. We 
have argued in Q.7 for the inclusion of a linked but separate data-led ‘Context’ section, which 
provides systematic intelligence about the inputs at the disposal of different institutions and 
submitting units. While ‘clustering’ (as used in the KEF) might be a step too far, we think that it is 
vital that Future Research Assessment takes more intelligent account of the variations in size, 
scale, mission and context of different HEls: to enable account to be taken of ‘what you do with 
what you are given / have’. 


Finally, we suggest that the decision for REF 2021 to incorporate the impact template in the 
Environment section should be reversed, and that a similar device to the impact template should 
be reinstated (for instance a narrative account of the underpinning approach to support impact, 
building perhaps on lessons learned from the use of narratives in the Local Growth and Public and 
Community Engagement in the KEF). This would contextualize the submitted impact case studies 
within the Institution’s / Unit’s wider impact and engagement approach. Anecdotally, it appears 
that few panels actually brought the impact section of the Environment statement to the 
attention of their impact assessors in REF 2021, potentially resulting in a significantly less rounded 
judgement. 
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Section four: assessment processes 
Frequency 


Outcomes from the REF are used to inform the allocation of block grant funding to universities. This 
funding method ensures a degree of research stability and independence not provided by other 
funding sources, because the results of research assessment are used over a prolonged period and 
the funding can be used as providers choose rather than being directed to particular research 
programmes. 


Participants at the roundtables were asked to consider the frequency and sequencing of assessment 
exercises. Currently, the REF takes place every 5-7 years and assessment of all disciplines takes place 
in parallel. It has been suggested that a more regular exercise could increase its formative element 
and would ensure that funding based on REF outcomes more accurately reflects recent performance. 
However, it was noted that this must be weighed up against the potentially destabilising effect 
arising from the uncertainty of funding outcomes on a more regular basis. 


The funding bodies recognise that views on the frequency of a future exercise will depend on the 
overall design of the assessment system. For example, it would not be feasible to run the exercise as 
it currently stands every three years without significantly increasing the burden on the sector. 
However, the funding bodies are keen to understand in principle whether the sector considers the 
availability of more current information to be more important than the stability offered by a less 
frequent exercise. 


17. When considering the frequency of a future exercise, should the funding bodies prioritise: 


a) stability 


b) currency of information 


c) botha. and b. Yes 


d) neither a. nor b. 


e) Don’t know. 


18. Do you have any further comments to make regarding the prioritisation of stability vs. 
currency of information? 


Building on our earlier comments, we would like to see intelligent alignment between the 
Environment component of future research assessment and other instruments / requirements on 
HEIs, to minimise burden and increase coherence. For example, data linked to the various 
Concordats could be gathered incrementally and the future REF could provide a summative 
assessment of the overall performance every 5 to 7 years. 


It will also be important to articulate the relationship between data collected for the KEF and for 
the REF, given the significant overlaps. 
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Sequencing 


During discussions on the frequency of the exercise, some roundtable participants expressed some 
appetite for moving to a rolling exercise, sequenced by main panel or by assessment element. It was 
suggested that this would remove some of the perverse behaviours linked to the cyclical nature of the 
REF, particularly around recruitment and publishing practices. It was suggested that this would also 
reduce burden at an institutional level as effort would be spread across a number of years, rather 
than focused ona single end point. As with the frequency of the exercise, any decision to move to a 
rolling exercise must be weighed up the potentially destabilising effect of such a change. 


19. Should a future exercise take place on a rolling basis? 


a) Yes, split by main panel 

b) Yes, split by assessment element (e.g. 
outputs, impact, environment) 

c) No 

d) Don’t know. X 


20. Do you have any further comments to make regarding conducting future research 
assessment exercises on a rolling basis? 


There is an argument that periodic ‘immersions’ in the REF work well — providing time in between 
to consolidate learning; and to maximise learning across the whole system. But does that actually 
happen, or does the REF become a ‘cliff edge’, with activity falling off in between the iterations of 
the exercise? 


It would be interesting to explore a different ‘rhythm’, and a more incremental process of 
evolution (though this might not always show advancement). The reporting could be rolling with a 
summative exercise every five years as a ‘reset’. 


We also suggest, as a ‘thought experiment’, the potential benefits of decoupling the assessment 
of outputs and impact, and perhaps sequencing these exercises so that they don’t happen 
simultaneously. 


If this were implemented, the assessment of impact could be delivered through a ‘lighter touch’ 
approach, with perhaps four broad discipline-linked panels and a fifth interdisciplinary panel. 
Each would be chaired by an impact specialist, and largely made up of impact specialists with a 
small number of ‘Research assessors’ to assure that the underpinning research meets the 
threshold. 


This would have a number of advantages: 
e Rebalancing the exercise so that Impact and Outputs have equivalent status 
e Strengthening the role of ‘users’ and stakeholders in research 
e More effectively bringing impact specialism into the review process 
e Better networking and connecting the impact expertise across the exercise 
e Reducing some of the overheads of the exercise 


We are also aware of the downsides of this approach: 
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It risks fuelling a two-tier approach in institutions 
It risks downplaying the vital interaction between research and impact, and silo-ing 
the two processes 


A related, but less radical suggestion, would be to consider moving to the ‘co-chairing’ of panels, 
with an academic and ‘user’ expert sharing responsibility for the assessment process. 
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Granularity 


A number of the changes made between REF2014 and 2021 were intended to reduce the emphasis 
on the individual in order to shift the focus onto the submitting unit as a whole. At the same time, an 
institutional-level environment statement is being piloted alongside REF 2021. 


Roundtable discussions emphatically rejected a return to a more individual-focused exercise. 
However, views were divided on the extent to which future exercises should retain the Unit of 
Assessment structure. Those in favour of a more institution-focused approach frequently cited the 
current (perception of) disadvantage to inter- and transdisciplinary research in a discipline-based 
system. It was also noted that many of the issues relating to research culture and environment can 
only be addressed at the level of the institution. 


There was, however, concern amongst others that a move to a fully institutional-level assessment 
would conceal the ‘pockets of excellence’, particularly in less research-intensive HEIs, and would 
make REF assessment outcomes less robust indicators of excellence. This is an important 
consideration for the funding bodies, given the continued link with funding. 


21. At what level of granularity should research be assessed in future exercises? 


a) Individual 


b) Unit of Assessment based on disciplinary areas 


c) Unit of Assessment based on self-defined research themes 


d) Institution 


e) Combination of b. and d. Yes 


f) Combination of c. and d. 


g) Other (please specify) 


22. Do you have any further comments to make regarding the granularity of assessment ina 
future research assessment exercise? 


A mix of institutional and ‘local’ assessment seems the most sensible option. It provides useful 
intelligence around the state of different fields and the current system accommodates self- 
defined research themes alongside disciplinary expertise 
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Metrics 


Roundtable discussions suggest limited appetite for increasing the role of metrics in the assessment 
of outputs. However, there was greater support for exploring quantitative indicators in the 
environment section. The use of metrics in the REF has been discussed at length, most notably in the 
2015 Metric Tide Report. However, discussions persist in the sector 


23. To what extent and for what purpose(s) should quantitative indicators be used in future 
assessment exercises? (Please select as many as apply) 


a) Move to an entirely metrics-based assessment 


b) Replace peer review with standardised metrics for: 
e Outputs 
e Impact 
e Environment 


c) Use standardised metrics to inform peer review of: 
e Outputs 
e = Impact 
e = Environment 

d) Should not be used at all. 


e) Other (please specify) YES — see below 


24. Do you have any further comments to make regarding the use of metrics in a future 
research assessment exercise? 


We want to highlight significant issues with how the REF treats EDI in relation to the assessment 
of impact, and argue that major changes are required to address this, in line with the Public 
Sector Equality Duty which puts the onus on organisations to positively promote equality, not just 
avoid discrimination. 


The Duty sets out that organisations need to demonstrate that they are thinking about how they 
can positively contribute to the advancement of equality and good relations. It requires equality 
considerations to be reflected in the design of policies and the delivery of services, including 
internal policies, and for these issues to be kept under review. Other funders have embedded 
these requirements, for instance, Arts Council England require their National Portfolio 
Organisations to develop Equality Action Plans. These action plans encompass both workforce 
development (‘internal’ polices and processes) and the external engagement / programming / 
partnership work that the organisation delivers. Inclusion has been a focus for the Heritage 
Community Fund for many years and in 2019 they adopted a mandatory inclusion outcome for 


grants and an updated Workforce Equality Statement. 


This area requires urgent and systematic review. We note: 


e REF Impact guidance provides no explicit guidance or expectations about EDI in relation 
to the beneficiaries of research, or its impact outside academia, although the definition of 
‘reach’ invites information about the diversity of beneficiaries 

e The KEF captures no intelligence about EDI in relation to the outcomes achieved or the 
communities involved and benefiting 

e The KE Concordat makes passing reference to EDI, but only in very generic terms 
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e  HEIF strategies do not require any focus on EDI in relation to the beneficiaries of KE or the 
partnerships that will be entered into 

e UKRI’s tool to collect data about the outputs, outcomes and impacts of research, 
Research Fish, gathers no data about the EDI characteristics of beneficiaries or partners 


Burden 


The cost and bureaucratic burden of the REF are frequently cited in criticism of the exercise. 
Roundtable discussions identified some sources of burden specific to the current exercise (e.g. special 
circumstances procedures), along with the overall scale and complexity of the exercise. However, 
several participants stated that the bureaucracy is, to a certain extent, generated by institutions’ 
approaches to the REF and can be difficult to distinguish from activities that would be carried out as 
part of business as usual or in response to requirements elsewhere in the system (e.g. by research 
funders). Several respondents expressed scepticism that burden would increase or diminish 
significantly with changes made to the exercise. It was also noted that changes may, in themselves, 
create additional burden for institutions regardless of their nature or intent. While UK-wide research 
assessment falls outside the scope of the ongoing ‘Independent review of research bureaucracy’ 
commissioned by the Department for Business, Energy & Industrial Strategy, related discussions have 
largely echoed the views expressed in the roundtables. 


25. How might a future UK research assessment exercise ensure that the bureaucratic burden 
on individuals and institutions is proportionate? 


There are a host of ‘moving parts’ that need to be taken into account in addressing this challenge, 
which we outline elsewhere in our submission. 


We recommend that a ‘system design’ approach be used to bring these different parts into more 
intelligent alignment. We note that such an approach has been recommended in the ITT for the 
second phase of the Concordats and Agreement review. 


We recommend that the 6 principles underpinning the Research Bureaucracy review be used to 
scaffold these discussions: 


e Harmonisation 
e Simplification 
e — Proportionality 


e Flexibility 
e Transparency 
e Fairness 


e Sustainability 


We also propose including an additional principle: Inclusion — to explore the intended and 
unintended impacts that might be realised with respect to EDI. 


Finally, we want to stress the importance of timeliness. The sooner that changes can be agreed 
and communicated the better, to reduce the time spent working ‘blind’, which contributes to the 
burden. 
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